It's about refinement, that's it. Z's are great cars, but they are rough around the edges. Do you want to sacrifice refinement for a car that has a coffee grinder for a motor and drivetrain? Personally, I love Z's but for example they are very annoying with a light flywheel because of the excess bullshit feeling you get when driving one. Jumping from any BMW to that is like shifting butter to sitting on one of those vibrating mats, the shift feel is meh, the vibrating shifter is abnoxious and the sound the engine makes is faux to me. One day on a track, hell yes, everyday, give me car with more R&D.
If your concerns are all out speed, if you love to flash your hazerds at hondas, make noise and look fast even though your chassis isn't that great then by all means GET the Z, that's the car for you.
Honestly, if it were me i'd upgrade your car... Yes upgrades are expensive, but for example a lightweight flywheel, U.D pullies, intake, reflash on that car will get you an easy .5 off your 1/4 mile E.T (i'd say low 15, high 14 if you can drive i.e 1.9 60 footers). Suddenly that 184hp car isn't so slow.. Still not fast enough? The centrifugal supercharger and twin screw kits are great and can be found used quite easily. Expect around 250-270whp at only 5-7lbs.